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H' 

r^ ' In the linear random eflects model, when distributional assumptions such as normality of the 

• I error variables cannot be justified, moments may serve as alternatives to describe relevant dis- 

i^H , tributions in neighborhoods of their means. Generally, estimators may be obtained as solutions 

j^ ' of estimating equations. It turns out that there may be several equations, each of them leading 

Cj , to consistent estimators, in which case finding the efficient estimator becomes a crucial problem. 

I I . In this paper, we systematically study estimation of moments of the errors and random effects 

in linear mixed models. 

►-^ ' Keywords: asymptotic normality; linear mixed model; moment estimator 

^ ■ 1. Introduction 

^J ' Normality or, more generally, the existence of a parametric structure on the distribution 

f^ I of random effects is a routine assumption for linear mixed models. In such a case, both 

^SJ ' the maximum likelihood estimator (MLE) and the restricted maximum likelihood esti- 

T-H I mator (RMLE) work well. Moreover, they are standard outputs in statistical software 

^ ■ packages such as SAS and R. A comprehensive account of the methodology is contained 

•^ I in the monograph of Verbeke and Molenberghs [7]. In recent years, more efforts were 

rxj . devoted to relaxing this assumption and using semiparametric or nonparametric meth- 

^ I ods to estimate the parameters of interest. Zhang and Davidian [9] suggested using the 

" ■ ■ scminonparametric representation of Gallant and Nychka [4] to approximate the ran- 

dom effect density in order to estimate parameters for linear mixed models. Cui, Ng and 
Zhu [3] used the estimation of moments in mixed effect models with errors in variables. 
Rank estimation was applied by Wang and Zhu [8] to estimate fixed effects. 

However, the aforementioned papers do not consider the estimation of higher moments 
that are useful for hypothesis testing and interval estimation for the parameters in the 
models. To the best of our knowledge. Cox and Hall [2] is the only reference in the 
literature that defines and studies the estimators of the errors and random effects for 
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higher than second moments. The authors of that work obtained the cumulants of the 
two components of variance based on homogeneous polynomials in a simple random 
effects location model, which is the sum of the one-level random effect and the error. For 
this model, Hall and Yao [5] studied nonparamctric estimation of the distributions of the 
errors and the random effects via empirical cumulant generating functions. To the best 
of our knowledge, no paper has investigated this issue for the linear mixed model under 
consideration. 

The contents of this paper are as follows: 

• In Section 2.1 we introduce the linear mixed model and derive basic properties of 
the generalized least squares estimator under weak conditions on the group sizes and 
the design variables. The fundamental Lemma 2.1 yields representations of certain 
polynomial functions of the overall errors in terms of individual and group errors. 
This will be the basic tool to answer a question posed by Cox and Hall [2] in the 
context of the simple random effects location model, namely, how to properly weight 
and combine certain polynomial functions of the residuals. 

• As a warmup, in Section 2.2, we consider the estimation of second moments. It turns 
out that by a proper combination of polynomial functions of the residuals, we can 
obtain second moment estimators which are asymptotically normal and have the 
same limit variance as if the unknown errors were known. 

• For third and fourth moments, the situation is more complex. In Sections 2.3 and 2.4, 
we propose and study estimators yielding efficiency and asymptotic normality under 
weak conditions on the design and group sizes. 

• As an alternative, in Sections 3.1 and 3.2, we study an extension of an estimator due 
to Cox and Hall [2] which may therefore be considered as a first step estimator. When 
the group sizes are all equal, our estimators have similar asymptotic properties to 
theirs. We show that for unequal group sizes, the obtained estimators may converge 
at slower rates unless some restrictive regularity assumptions are satisfied. 

• Section 4 presents some simulation studies, while proofs are deferred to the 
Appendix. 

2. Minimum variance estimation of moments 
2.1. Motivation and first results 

Assume that data are available from a linear mixed model, that is, we observe pairs 
{xij,yij),l <i<n,l<j <li, satisfying 

y^j =a + x'f^l3 + bi + eij. (2.1) 

Here, i denotes the group index, while the measurements within this group are indexed 
by j. The integer li is the sample size within group i. The row vector x^ is a p-dimensional 
input vector corresponding to the jth observation in the ith group leading to the out- 
put yij. The relation between Xij and yij described by (2.1) contains the intercept pa- 
rameter a, the fixed effect regression parameter /3 and the one-level random effect bi for 
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group i, all unknown. Moreover, these quantities are disturbed by random errors Sij. It 
is assumed throughout that 61, . . . , 6„ are independent and identically distributed (i.i.d.) 
and also independent of all e^ , which are also i.i.d. Finally, we may assume without loss 
of generality that 

E&, =0 and Ee„- = ioi 1 <i <n and 1 < j <k. (2.2) 

Otherwise, we may incorporate unknown nonzero expectations in the intercept a. Let 7^ 
and 7^ denote the fcth moments of the random effects and errors, respectively. In this 
paper, we shall construct and analyze estimators of a, /?, 7^ and 7^, fc = 2,3,4, that 
are based on various estimating equations. These equations are obtained from proper 
nonlinear combinations of the residuals. For these, we first have to estimate /3 and a via 
a generalized least squares method. In the model (2.1), this leads to 



and 



Here, 



while 
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-. n li 
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denote the corresponding group averages. Furthermore, we let 

n 
i=\ 

the overall sample size. 

Theorem 2.1. Assume that the following conditions (2.6)-(2.8) are satisfied: 



lim E„ = S for some positive definite p x p matrix E; (2-6) 

n— ^C30 

1 " 1 

— /.-J 5-0 and N/n^ 00 as n—>cx3; (2-7) 
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Then, in distribution, we have 

7Vi/2(/3-/3)^AAp(0,7eS-^) 
and 



(2.9) 



(2.10) 



The estimators /? and a and their distributional behavior play an important role for 
motivating the estimation of 7^' and 7^ since this will be based on the residuals 
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In view of (2.4), we have 



d-a = 6 + £-x'(^-/3). 



from which it follows that 



= {b, -b) + (£.y - e) + {x,, - S)'(/3 ~ /§) 
= (6,;+e,,)-(6 + £-) + z,',(/3-/3). 



(2.11) 



Set 



■bi^Sij, 



a sum of two independent zero-mean random variables. 

When the /^'s are equal and /3 = 0, that is, in the simple random effects location 
model, Cox and Hall [2] used homogeneous polynomial functions to construct estimating 
equations. In the present paper, we consider more general situations in which new special 
nonlinear functions of the e^j 's are important tools to derive estimating equations for 

7^=Efe^ and 7^=E4- 
For this, define, for 1 <i <n and 1 < 771 < fc. 



/™w=E«" 



i=i 



E' 



k—Tn 



The following lemma turns out to be crucial for our analysis. 
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Lemma 2.1. We have 

k tAm 

t=0 s=(t-fe+m)VO 




E4 E' 
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\]=i 



Here, a/\h and a\/b denote the minimum and maximum, repectively, oj two real numbers a 
and b. 

The proof follows froni simple arithmetic. When we take expectations, usually many 
of the terms in the expansion of /^(?) will vanish, mainly because the £y 's and bi's are 
centered and independent; see (2.2). Moreover, by taking proper linear combinations of 
the /^(z)'s, we shall be able to represent the 7^'s and 7^'s in terms of the /'s. These 
so-called estimating equations will then lead to associated estimators. 

For example, in the case of 7^, we have 



hf2ii)'fn^)-hj24 



E^ 



from which it follows that 



E[kfi{t)-ff{t)]=k{k~lhl 



This equation does not incorporate any 6-term, so it may serve as a basis for the estima- 
tion of 7^. For moments 7^ and 7^, fc > 2, things become more delicate. At first, it is not 
clear how to combine the /^(i)'s in order to get efficient estimators. This issue is dealt 
with in Sections 2.2-2.4, for fc = 2,3 and 4, respectively. In Section 3, we briefiy discuss 
the extension of Cox and Hall [2] to the regression case and show that it may cause some 
inefficiencies. 

Remark 2.1. We only remark in passing that the results of this and the following 
sections may be extended to group sizes Ini, 1 < i < ?i, that is, when the Vs depend on 
the number n of groups and therefore form a triangular array. 

2.2. Estimation of 7^ and 7^ 

We start by estimating 7^ and 7^. As mentioned above, 

E[l,fi{z)-f?{z)]^l,ih-l)^l 

Averaging over 1 < i < n and replacing the unknown e's by the residuals leads to the 
estimator 



ll~ 
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Similarly, the equation 
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E[fU^)-f|{^)]^h{h~lh! 
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leads to the estimator 



ii 



^t^{(p-') -P 



Theorem 2.2. Under the conditions of Theorem 2.1, when 7^ and '^'^ are finite, we have 
that 



and 



where 






(2.12) 
(2.13) 
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fJ-e=le- he ) "■'"■d fJ-b = lb - (lb ) 

It is interesting to note that (2.12) and (2.13) will be shown by verifying 



and 



7VV2[7,^~7e^]=^-^/^ 



l/2r-2 2i -'- 



' n li 



+ Op(l) 



Ei^l-ii) 



-Op(l). 



In other words, 7^ and 7^ are as efficient as the moment estimators based on the true 
(but unknown) Eij and bi. 

2.3. Estimation of 7^ and 7^ 

In this section we show how to estimate 7^ and 7^ with minimal variance. Again, this 
may be achieved by properly combining the /^j(i)'s. From Lemma 2.1, we obtain 

¥.fiii) = ia! + hil 



and 



Wc conclude that 



m{i) = itrb+hit- 



E[2/i^(z) + Iffliz) - 3kfi{t)] = kik - l)ih - 2)^i. 
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The corresponding estimator of 7^ becomes 



For 7^, the relevant equation is 
leading to the estimator 



V 
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Theorem 2.3. Under the conditions of Theorem 2.1, when 7^ and^^ are finite, we have 
that 

and 

where 

As for second moments, these quantities denote the minimum variances, which may 
be achieved for empirical estimators based on the true Sij and bt, respectively. 

2.4. Estimation of 7^ and 7^ 

For 7^, we are also looking for a combination of /^'s such that the expectations include 7^ 
but no other moments. First, from Lemma 2.1, we have 



E/2'(0 = nit + in + ^iDihl + k{k i){il? + kit 

and 



^m = itit + Qihhl + ^ 
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Finally, we put 



fii^)- 



-\ 2 






Clearly, 



j = l A;=l j = l k=l 



ik) 



(2.14) 



= Iht + hit + (2^' + 4/.)7'7s + {ll - '.)(7e )'• 
We now combine these expressions in a proper way. In particular, we check that 

nil} - 21, + 3Kkf!{i) ~ 4/|(z)) + %U!^{i) - ■iHii) - 3(2Z, - 3)/|(*)] 
= /,(/, -l)G,-2)G,-3)7^ 

At first sight, the coefficients may look a little strange, but they appear as solutions of 
linear equations incorporating E/f , . . . , E/| such that all terms involving moments other 
than 7^ vanish. Our minimum variance estimator of 7^ thus becomes 



n 

y4 = iV-iV 

^' ^iU-DiU- 



^G,-l)(/,-2)(/,-3) 
X J [If - 21, + 3) 
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E' 



For 7^, the relevant equation is 



IE[/i'(0 - 6/2'(*) + 8/3'(z) - 6/i(z) + 3f^{i)] = /.(/. - 1)(;, - 2)(/, - 3)7,^ 



givmg us 



it- 



n 

-Y - 



li 
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Theorem 2.4. Under the conditions of Theorem 2.1, when j^ and j^ are finite, we have 
that 

and 

where 

and 

As in previous cases, //^ and /i^ are minimal variances. 

3. First step estimation 
3.1. Estimation of 7^ and 7^ 

In this section, wc briefly discuss the fact that different choices of estimating equations 
may lead to inefficiencies. These observations eventually lead us to the efRcient estimators 
discussed in the previous section. For the third moments, recall that 

E/l i^) = hi! + kll and E/| {€) = l^l + hll , 
from which 

Summation over 1 <i <n yields 

3_Er=i(i/a»-i))^E/i«-E/i«] 



7e 



N 



If we replace the expectations by their sample analogs and the true e's by the residuals, 
then we come up with an estimator of 7^ similar to that of Cox and Hall [2], where all Zj's 
are equal and there are no covariate effects: 



^_. E„.(i/«. -!))[<. Et. 4 -(Et.^5)(Et.^,)l 



N 
In the same way, we obtain 



7^^ 



^ n ^ ( li h h ^ 
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To formulate limit results for 7*"' and 7^ , we recall that 



2\3 



and put 



Here, 



M^=7e^-(7eT" 67,^,^ + 9(7,^) 
Hf = it - ill? Hit + (4c + 5)(7,^)3 + 4(7,^)3x^E-ixo. 

n I, 



^:=A^-^EE^ 



and (as before) 



The vector xq in ^* equals 



i=i j=i 



^ n ^ li 

1=1 j=i 



xq = lim (x* — X' 



n ■^nli 



while 

N " 
c= lim ^5-y^L~\ 

n— ^00 Ti -*- — ^ 

assuming that both limits exist. 

A detailed qualitative interpretation of these quantities will be deferred to the end of 
this section. 

Theorem 3.1. Under the conditions of Theorem, 2.1, when 7^ andj^ are finite, we have 
that 



where 



N'/^Te' - 7e') ^M(0, ^f + 47,^(7,^ - (1 - d){^lf)), 

En 72 v^n 7 

1=1 'i Z^j=l '■I- 



(3.1) 



d = lim 



TV 



lb 



n'/^[rb'-l!]^Ml{0,^il) asn- 



(3.2) 



Remark 3.1. As in Section 2, the estimator in the 6-case achieves the minimum vari- 
ance. It equals the variance of the moment estimator based on the true but unknown hi. 
In the e-case, things are less transparent. For example, assume that Ini = /° are all equal 
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for l<i <n, a situation studied by Cox and Hall [2]. If l^^ -^ oo, then c = 1, xq = and 
d = 0. Hence, 

the variance of the (central) moment estimator based on the true e^j . The total variance 
therefore becomes 

which, by the Cauchy-Schwarz inequality, exceeds ^^. Hence, in this situation, 7*'^ is 
inefficient. 

Remark 3.2. If U = i"" with < a < 1, then c = 1/(1 — a?) > 1 becomes large as a — >■ 1. 
Hence, the quality of the Cox-Hall-type estimator deteriorates in such situations. Worse 
than that, as our proofs reveal, asymptotic normality may fail in situations where the 
limit d is not finite. 

3.2. Estimation of 7^ and 7^ 

For fourth moments, taking expectations of f^{i) and ,f^{i), we again obtain 

E/4 (*) = hit + Qhlhe + H (3.3) 

and 

E/3 (0 = Iht + ik{k + l)lWe + hit, (3.4) 



from which it follows that 



4 E/|(z)-E/|(») 
^b = TTi n 37^7^ . 



k{h-l) 
Averaging over 1 < z < n leads to the estimator 



-Ei;(i-i)|E4E^^.-E4|-37? 

2=1 \.] = l J = l J = l ) 



where 7^ and 7^ were studied in Section 2.2. From (3.3) and (3.4) we immediately obtain 

4 ^,E/|(»)-E/|(^) 22 

hie = T"^ *^^ ^^ 



and therefore to 



-^-^Er^^»E4-E4E^4 



Cox and Hall [2] also considered these estimators; however, we have discovered that the 
limit variances are larger than those given in their paper. Therefore, we propose the 
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following modification. First, recall that 

E/K*) = Iht + {ll + ^iDlWe + hiU - l)(7e)' + hit- (3.5) 

In addition to the fm{i) with vi < k, we again need 



fH^)- 



-\ 2 






It follows from (3.5) and (2.14) that 

nf2i^) im - ('f - I'ht + (If + 3/f - ihhhi 



(3.6) 



To estimate 7^, we are looking for a linear combination of (3.3), (3.4) and (3.6) so that 
the terms 7jJ and 7^7^ cancel out. In fact, it is easily seen that 

E[(2/2 _ l,)f^(^) - (5h - 4)/|(*) + Sf^ii) - SfHi)] ^ 2k{k - l){k - 2)7'- 

The corresponding estimator of 7^ becomes 

( li U li 



if- 



n-^y: 



1 



2(/,-l)(/,-2) 



(2Z?-Z,)5^4-(5Z.-4)^45^e, 



j=i 



j=i j=i 



3 E4 E' 



^j=i 



w=i 



E' 

U=i 



Following this idea, we also get an estimator of 7^. Subtracting (3.3) from (3.4), we 
obtain 

E[/|(*) - ./4 (»)] = ill - k)lt + 3(^1 - k)llll 
Together with (3.6), this yields 

3E[/|(*) - flii)] - (Z, +4)E[/|(*) - ft(i)] = 2k{k - l){k - 2)7,4 

and therefore 

( / h \ / h \^ / h \^ 



1 

7r--E 

T7 ^ ^ 



1 



^,,,_,,^iHg^"Jig-J -%g^l 



('■+<)(i:4)(i;««)+<'.+«i:«d 



In the following theorem, we summarize the main results on the limit distributions of 7** 
and 7^- 
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Theorem 3.2. Under the conditions of Theorem 3.1, when j^ and j^ are finite, we have 
that 

N'/'{%' - ^t)^mo. m:^ + hivit - (1 - d){ii? Hit + ^hff]) 

and 

where, again, 

f^t = i!'iitr-»i!i! + iGiKi!r 

and 

A' = il - (%')^ - Hil + (fc + fHh'ef + h'h!r4^-'^o. 

Remark 3.3. Our earlier Remarks 3.1 and 3.2 also apply to fourth moments. This more 
or less led us to look for the new estimators studied in Section 2. 

Remark 3.4- We will now discuss the results of this paper in a qualitative way. Suppose 
that all the 6i's and e^j's are known to the observer. Then, rather than computing resid- 
uals, they could be used directly to nonparametrically estimate the (central) moments of 
interest. Simple computations then show that the variances of these estimators equal /i^ 
and /ig, respectively. In the case where only residuals are available, the improper weight- 
ing in 7**^ yields variances which heavily depend on the design (via XqJ^~^xq), the group 
sizes (via the constants c and d) and the noise variables bi (via 7^). In such a situation. 
Theorems 2.3 and 2.4 provide new estimators which also attain the minimum variance 
in the £-case and are less vulnerable to the model design. 

4. Simulation study 

To demonstrate the usefulness of our estimation procedures, a small simulation study 
will be carried out. The data sets are generated from the model (2.1) with a = 1 and 
/3 = (1, 2)'. To estimate the model parameters and the third and fourth moments using the 
methods developed in this paper, the group values are randomly drawn from a Poisson 
distribution with mean 5. The design matrices are generated from a zero- mean normal 
distribution with covariance matrix (n a \) • For the random effects bi and the errors e^, 
we consider the following five cases: 

.i.d. 0.5^(0, 1) and 6, -i.i.d. 0.5^(0, 1); 
.i.d. 0.5^(0, 1) and b, -i.i.d. 0.5f(8); 

i.d. 0.57Vi(0, 1) and b^ ~i.i.d. 0.5r(l, 1) - 0.5; 

i.d. 0.5i(8) and bi -i.i.d. 0.5t(8); 

i.d. 0.5f(8) and bi -i.j.d. 0.5r(l, 1) - 0.5. 



(a) Eij 

(b) Eij 

(c) Eij 

(d) e.j 

(e) Sij 



The true values of the 2nd-4th moments of the errors and random effects are given in 
Table 1. A/i, F and t correspond to the normal, gamma and t distributions, respectively. 



14 P. Wu, W. Stute and L.-X. Zhu 

Table 1. The true values of the 2nd-4th moments of 
the random and group errors 



c.d.f 2nd 3rd 4th 

0.5iV(0,l) 0.25 0.1875 

0.5t(8) 0.333 0.5 

0.5r(l,l)-0.5 0.25 0.25 0.5625 



The following simulation results are based on 1000 samples of data {(xyjj/y) :i = 
1, . . . ,n,j = 1, . . . ,li} with n = 50, 100. The estimated mean, standard deviation and root 
mean squared error of the estimators suggested above are reported in Tables 2 and 3. 
Table 2 presents the results for the model parameters and second moments. For the 
purposes of comparison, we also include the results for the MLE. Table 3 presents the 
results for the minimum variance estimators of the third and fourth moments. 

In Table 2, the comparison with the MLE shows that our estimators are very competi- 
tive, although such a comparison is actually in favor of the MLE when we assume that the 
distribution is parametric. In fact, empirical studies in the literature also show that the 
assumption concerning the distribution of the random effects hardly influences the pa- 
rameter estimates; see Butler and Louis [1] and Verbeke and Lesaffre [6] for details. This 
indicates that the estimation of moments for the model parameters performs very well. 



Appendix 



Proof of Theorem 2.1. We first study /3. It follows from (2.1) and (2.3) that 

-| 71 I i 

/3-/3= ^n V^S,7^(Xij--Xi.)£jj-, (5.1) 

where E„ is given in (2.5). To show (2.9), we fix a e M^. It suffices to prove that 

N^^^a'{(3 - /3) ^7Vi(0,7^a'S"ia) in distribution. 

Since, according to (5.1), /3 — /3 is a sum of zero-mean independent random vectors, 
it remains to check the variance and verify Lindeberg's condition. The variance of 
N^/^a'0-P) equals 

2 n li 

by (2.6). To verify Lindeberg's condition, we first fix 5 > 0. The Lindeberg function then 
equals 

L„(5)=7V-i^^[a'S,Ti(.T,;,-x..)p / 4dP. 

,^1 , = 1 J{|a'S-i(^,,-S..)l|e..l><SVSl7} 
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Table 2. The results for a, /3, 7? and 75 in cases (a) 
to the MLE) 



(e) (the numbers in brackets correspond 



Case n Result a 



Pi 



P2 



ll 



ll 



(a) 



(b) 



(c) 



(d) 



(e) 



50 mean 
std 
rmse 
100 mean 
std 
rmse 

50 mean 
std 
rmse 
100 mean 
std 
rmse 

50 mean 
std 
rmse 
100 mean 
std 
rmse 

50 mean 
std 
rmse 
100 mean 
std 
rmse 

50 mean 
std 
rmse 
100 mean 
std 
rmse 



1.0004 
0.0748 
0.0748 
0.9965 
0.0544 
0.0545 
1.0022 
0.0860 
0.0861 
1.0026 
0.0598 
0.0598 

0.9988 
0.0754 
0.0754 
0.9974 
0.0527 
0.0527 

1.0008 
0.0843 
0.0843 
1.0004 
0.0618 
0.0618 

1.0025 
0.0670 
0.0670 
1.0007 
0.0537 
0.0537 



(1.0005 


0.9986 


0.9984 


2.0009 


2.0012 


0.2497 


0.2480 


0.2451 ( 


(0.0749 


0.0428 


0.0424 


0.0430 


0.0427 


0.0177 


0.0177 


0.0550 ( 


(0.0749 


0.0428 


0.0424 


0.0430 


0.0427 


0.0178 


0.0178 


0.0552 ( 


(0.9964 


1.0000 


0.9999 


1.9995 


1.9996 


0.2496 


0.2496 


0.2476 ( 


(0.0544 


0.0307 


0.0304 


0.0302 


0.0300 


0.0128 


0.0128 


0.0392 ( 


(0.0546 


0.0307 


0.0304 


0.0302 


0.0300 


0.0128 


0.0128 


0.0393 ( 


(1.0023 


1.0004 


1.0002 


2.0010 


2.0014 


0.2490 


0.2490 


0.3301 ( 


(0.0860 


0.0438 


0.0438 


0.0427 


0.0427 


0.0184 


0.0184 


0.0978 ( 


(0.0860 


0.0438 


0.0438 


0.0427 


0.0427 


0.0185 


0.0184 


0.0979 ( 


(1.0026 


0.9991 


0.9991 


2.0005 


2.0004 


0.2489 


0.2489 


0.3322 ( 


(0.0598 


0.0299 


0.0296 


0.0299 


0.0296 


0.0128 


0.0127 


0.0652 ( 


(0.0598 


0.0299 


0.0296 


0.0299 


0.0296 


0.0128 


0.0128 


0.0652 ( 


(0.9988 


1.0004 


1.0003 


1.9962 


1.9960 


0.2485 


0.2485 


0.2421 ( 


(0.0754 


0.0430 


0.0426 


0.0418 


0.0415 


0.0178 


0.0178 


0.0981 ( 


(0.0754 


0.0430 


0.0426 


0.0420 


0.0417 


0.0179 


0.0179 


0.0984 ( 


(0.9974 


0.9993 


0.9993 


2.0005 


2.0007 


0.2481 


0.2481 


0.2437 ( 


(0.0528 


0.0298 


0.0294 


0.0302 


0.0298 


0.0121 


0.0121 


0.0731 ( 


(0.0528 


0.0298 


0.0294 


0.0302 


0.0298 


0.0123 


0.0123 


0.0733 ( 


(1.0007 


0.9992 


0.9992 


2.0001 


2.0011 


0.3298 


0.3297 


0.3239 ( 


(0.0842 


0.0479 


0.0474 


0.0484 


0.0479 


0.0294 


0.0294 


0.0896 ( 


(0.0842 


0.0479 


0.0474 


0.0484 


0.0480 


0.0296 


0.0296 


0.0900 ( 


(1.0003 


1.0006 


1.0004 


1.9996 


1.9996 


0.3315 


0.3315 


0.3305 ( 


(0.0618 


0.0336 


0.0334 


0.0351 


0.0350 


0.0213 


0.0213 


0.0697 ( 


(0.0618 


0.0336 


0.0334 


0.0351 


0.0350 


0.0214 


0.0214 


0.0697 ( 


(1.0023 


1.0016 


1.0012 


1.9977 


1.9977 


0.3311 


0.3313 


0.2433 ( 


(0.0670 


0.0488 


0.0488 


0.0494 


0.0492 


0.0297 


0.0298 


0.1010 ( 


(0.0671 


0.0489 


0.0488 


0.0495 


0.0493 


0.0297 


0.0298 


0.1012 ( 


(1.0006 


0.9994 


0.9993 


2.0011 


2.0010 


0.3323 


0.3323 


0.2446 ( 


(0.0536 


0.0348 


0.0344 


0.0335 


0.0330 


0.0220 


0.0220 


0.0708 ( 


(0.0536 


0.0348 


0.0344 


0.0335 


0.0330 


0.0220 


0.0220 


0.0710 ( 



(0.2449) 
(0.0549) 
(0.0552) 
(0.2473) 
(0.0393) 
(0.0394) 

(0.3296) 
(0.0978) 
(0.0979) 
(0.3319) 
(0.0651) 
(0.0651) 

(0.2418) 
(0.0981) 
(0.0984) 
(0.2440) 
(0.0730) 
(0.0733) 

(0.3232) 
(0.0894) 
(0.0900) 
(0.3303) 
(0.0697) 
(0.0697) 

(0.2425) 
(0.1014) 
(0.1017) 
(0.2442) 
(0.0707) 
(0.0709) 



Recall that, by (2.. 



^n — 



max,- 1 .T,-,- — Xi. 



Ari/2 



We conclude, by the Cauchy-Schwarz inequality and (2.6), that 



LniS)<a't-^a e^dP^O, 

./{||a'E;r^j||£|>5C^^} 



16 



P. Wu, W. Stute and L.-X. Zhu 



Table 3. The results for 7* and 7^ (fc = 3,4) in cases (a)-(e) 



Case 



Result 



7.^ 



1! 



it 



(a) 



(b) 



(c) 



(d) 



(e) 



50 



100 



50 



100 



50 



100 



50 



100 



50 



100 



mean 


-0.0004 


0.1852 


-0.0003 


0.1813 


std 


0.0170 


0.0325 


0.0519 


0.0974 


rmse 


0.0170 


0.0326 


0.0519 


0.0976 


mean 


-0.0002 


0.1867 


0.0005 


0.1835 


std 


0.0125 


0.0234 


0.0362 


0.0688 


rmse 


0.0125 


0.0234 


0.0362 


0.0689 


mean 


0.0002 


0.1866 


-0.0003 


0.5054 


std 


0.0171 


0.0331 


0.1948 


0.8577 


rmse 


0.0171 


0.0331 


0.1948 


0.8578 


mean 


0.0002 


0.1858 


0.0076 


0.5003 


std 


0.0121 


0.0239 


0.1276 


0.4835 


rmse 


0.0121 


0.0240 


0.1278 


0.4835 


mean 


0.0007 


0.1864 


0.2290 


0.4962 


std 


0.0174 


0.0335 


0.2235 


0.7649 


rmse 


0.0174 


0.0335 


0.2244 


0.7678 


mean 


0.0002 


0.1848 


0.2380 


0.5317 


std 


0.0123 


0.0216 


0.1778 


0.6445 


rmse 


0.0123 


0.0216 


0.1782 


0.6445 


mean 


0.0001 


0.4754 


0.0033 


0.4477 


std 


0.0557 


0.1832 


0.1520 


0.4510 


rmse 


0.0557 


0.1848 


0.1520 


0.4542 


mean 


0.0007 


0.4862 


0.0014 


0.4796 


std 


0.0403 


0.1611 


0.1098 


0.4100 


rmse 


0.0403 


0.1617 


0.1098 


0.4102 


mean 


-0.0011 


0.4832 


0.2278 


0.5007 


std 


0.0578 


0.2068 


0.2311 


0.8052 


rmse 


0.0578 


0.2075 


0.2322 


0.8075 


mean 


0.0001 


0.4979 


0.2385 


0.5355 


std 


0.0427 


0.1726 


0.1753 


0.6569 


rmse 


0.0427 


0.1726 


0.1757 


0.6575 



as required. This proves (2.9). For d, we immediately get from (2.4) that 

n ^ — ^ n ^ — ^ 



a — a : 



i=l 



i=l 



From (2.7), it follows that n^l'^e — ?► in squared mean and hence in probability. 
Furthermore, by (2.7)-(2.9), 



"1/2 ^ x',. (/§-/?) ^ in probability. 
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Summarizing, 

n 

This shows (2.10) and thereby completes the proof of Theorem 2.1. D 

Proof of Theorem 2.2. From the definition of 7^ and (2.11), we readily get 



n if'' 



U U "I 

- E E [^y + 4 (/^ - /3)] [e^fc + ^.'fe (/3 - /3)] 

J = l A;=l J 

n li n ^ n li 

EE4-Ei-^E^^.^-+2EE^^.4(/?-/^) 

Ei-^E(4(/3-/3))'-E;-3tEE[4(/3-/3)][4(/3-/3)] 
E7-tE-^^^-^^'c(/3-/3) 



i=l j^k 

= I-II + III + IV-V-VI. 

Of these six terms, only the first will be a leading term, while the others are remainders. 
For example, II is a sum of centered independent random variables with variance 



" 7 



i=l 

We conclude, in view of (2.7), that 

N'^^^II -^0 in probability. 
To show the same for III, it suffices to prove, because of (2.9), that 

N~ 2^, y^^ij^ij ^^0 in probability. 
Again, this is a sum of centered random vectors with covariance 



7f 



7V2 "^Iv 
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Similarly, the convergence of N ^^^IV, N ^/"^V and N ^/^ VI to zero follows from (2.6)- 
(2.9). All together, this shows that 



N'/'[f^~j^J=N-'/' 



n li 






-0p(l) 



and hence (2.12), by a simple application of the central limit theorem. To show (2.13), 
we note that 



"rl- 



1 " 1 



^ij^ik ■ 



=1 '^ * ' j^k 

Again using (2.11) and applying similar arguments to those used before, we obtain 



f, 



n ^ — ^ 



i=l 



from which it follows that (2.13) holds. 



n 



Proof of Theorem 2.3. We first deal with 7;^. Simple algebraic manipulations yield 



^^'4g^.(/,-i)(/.-2)J^;--^-^- 



Expanding iij into e^ — Sij + 6^ — (6 + e) + z[Aj3 — /3), wo may again neglect all contri- 
butions involving the z[Af3~ f3). Hence, up to an op(n^^") term. 



7.^ 



^ /.(/.- 1)(/.- 2) ^£;-^-^^' 



ly^ 1_ 

1 " 1 

1 " 1 

1 " 



Under the assumptions of the theorem, the first three sums are negligible, as are the last 
two terms. Hence, 

n 

1=1 
= n-'''Y.[b^-^l-Hh]+oAl). 
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The conclusion for 7;^ now readily follows from the central limit theorem. For 7^, we may 
write 



IZi-nII4«''+IIII' 



j^k i j 

If we once again ignore the higher order terms of 2^' (/3 — /?) , we find that in the expansion 
of 7^ , we have 

3 



Er4TE4^-+EE4+Mpy^/^) 



j#fc 



EE[4-37e%]+op(iV^/^). 



' 3 

The conclusion for 7^ now follows easily from the central limit theorem after centering 
theef^-. D 

Proof of Theorem 2.4. For 7^, we check that 



1 



^Tb / ^ 1 n T \ n o^/7 o\ /^ ^^J ^ik^il^im- 



j=tk^l=tm 

Again neglecting all terms that contain z[A(i — /3), we get, with Wj = 6, — (6 + e), 
n ^ 

"^^' = E ;,(Z,-l)(Z,-2)a,-3) .^E ^^.s^'c^^'^- 

1=1 j^k^l^m 

n 

+4E^^-^^E^^e.,..s. 

+ 6E^^E..e,.+4E|E-.- 

J=l jT^fc J=l J 



E 



vf. 
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Since the first four sums are all op(n^/^), we obtain 



^"'[lt-lt]=n-'/'Y.[<-^t]+or{l). 



The distributional convergence of 7Jj now readily follows from the central limit theorem 
after an expansion of the last sum into 



For 7I, we check that 



i=l 

;,4 



1 

n n ^ n _. _, 



j^k »=1 ^ ' ' ^ ' ' j^k^l 



1=1 j 



4- 



Similarly to the proof of Theorem 2.3, it can be shown that 

mt - it) = EE[4 - it - 47e^^d + or{N'/% 

i 3 

from which the conclusion follows. D 

Proof of Theorem 3.1. We first deal with 7^'^. By definition, 



«=4gis<;^i:s/"-- 



Our goal will be to use (2.11) in order to express the last double sum in terms of hi, Sij 
and negligible remainders. Actually, in view of (2.7)-(2.9), since the standardizing factor 
of -yl^ is 71^/^ = o(A^^/^), all terms in the expansion of 7^'^ containing z^(/3 — /3) are 
negligible. In other words. 



1 " 1 

^" = -EH;-3T)EE(^»+^'^--(^+^"))'(^»+=»'=-(^+^"))+°r(^^"'^')- 



;C,*3 _ 



j^k 
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After some simple but tedious rearrangem.ents, this becomes 
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76 ' 



1 " 1 r 

n S 1(1 -I) ^"^^^J ~ ^^''^''^ ^ ^^* ^ ^'^^''' ~ "^^^ + ^) X! ^^^^^' 



i=i '*('* 



S/fc 



J5^fc 



i#fe 



^ErkE(4-%^)+3(5f-7.^)E-. 



2—1 






j=l 



4 = 1 



To identify remainders, we note that ri,^/^(& + e) = Op(l). Also, all summands in the 
double and triple sums are centered and independent. Computation of variances shows 
that they are all negligible. In summary, we get 

n 

This i.i.d. representation of 7^'^ is the key tool for (3.2) - just apply the central limit 
theorem to the leading sum. 

We will only study 7*^ briefly. First, by definition, 

^7f=EE4-E^^f^- 



i=i j=i 



i=l 



To expand the two expressions into leading terms and remainders, recall that the final 
standardizing factor in (3.1) will be N^^'^, which is the same as in (2.9). We conclude 
that, under the conditions of the theorem, terms containing higher orders of zlJ/3 — (3) 
are negligible. Hence, up to remainders. 



EE^^s-EEi 



^+b,-b+zUl3-p)f 



i=i ]=i 



i=\ ]=i 



n li n li n li 

= EE4+EE34(-s"+^'-^)+EE3^'^(-^"+^'-^)' 

z— 1 J — 1 i— 1 i— 1 i— 1 :? — 1 

n li n li 

+ EE(-^+^^-^)' + EE344(/3-/3) 

i=l j=l i=l j=l 
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n li n li 

+ ^^3(-e + 5.-6)2z^(/3-/3)+^^6£,,(-e-+6.-6)z^(/3-/3). 

4=1 J = l 4=1 j=l 

A detailed study of these sums yields 

n ( h 

i=i I i=i 



3 _,3 1 ni, /_2 -;2\ 



e%-ri + 'ih{e 



- 7e (l + ^ + 2a;^S;;i (x,, - x,. )^ £,, 



-Op(l). 



The leading part is a sum of centered independent random variables to which the central 
limit theorem may be applied. Its variance satisfies 

E[£3 - ^lY + 472E[e2 - 72]2 + 4(7^)272^ 

- 67,2E£4 + (72)3 [5 + 4c + Ax'qY.-^xo\ + o(l) 

^Mf +47b'(7s'-(l-rf)(7e')'), 

as desired. This completes the proof of Theorem 3.1. D 

Proof of Theorem 3.2. The necessary arguments arc similar to those used before and 
are therefore omitted. Details may be obtained from the authors. D 

Acknowledgment 

This work was supported by Grant HKBU2030/07p from the Research Grants Council 
of Hong Kong. 

References 

[1] Butler, S.M. and Louis, T.A. (1992). Random effects models with non-parametric priors. 
Statist. Med. 11 1981-2000. 

[2] Cox, D.R. and Hall, P. (2002). Estimation in a simple random effects model with nonnormal 
distributions. Biometnka 89 831-840. MR1946513 

[3] Cui, H.J., Ng, K.W. and Zhu, L.X. (2004). Estimation in mixed effects model with errors in 
variables. J. Multiv. Anal. 91 53-73. MR2083904 

[4] Gallant, A.R. and Nychka, D.W. (1987). Semi-nonparametric maximum likelihood estima- 
tion. Econometrica 55 363-390. MR0882100 

[5] Hall, P. and Yao, Q. (2003). Inference in components of variance models with low replication. 
Ann. Statist. 31 414-441. MR1983536 



Linear mixed models 23 

[6] Verbeke, G. and Lesaffre, E. (1996). Large samples properties of the maximum likelihood 
estimators in linear mixed models with misspecified random-effects distributions. Tech- 
nical report, Biostatistical Centre for Clinical Trials, Catholic Univ. Leuven, Belgium. 

[7] Verbeke, G. and Molenberghs, G. (2000). Linear Mixed Models for Longitudinal Data. New 
York: Springer. MR1880596 

[8] Wang, Y.G. and Zhu, M. (2006). Rank-based regression for analysis of repeated measures. 
Biometrika 93 459-464. MR2278097 

[9] Zhang, D. and Davidian, M. (2001). Linear mixed models with flexible distributions of 
random effects for longitudinal data. Biometrics 57 795-802. MR1859815 

Received November 2009 and revised August 2010 



